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This article presents a control group study of the influence of a partnership-centered, community-based 
learning program on students’ academic writing. The improved writing of first-year students in the 
Chicago Civic Leadership Certificate Program { CCLCP ), we argue, results from the deeply situated 
learning that took place in the context of reciprocal, community-based relationships. We also argue that 
research on the impact of community-based learning should take into account the contemporary univer- 
sity’s emerging paradigm of engaged learning and research, which calls for a redefinition of partnership 
and reciprocity. 


Research in a Context of Partnership 

Calls for more rigorous research on the impact 
of service-learning suggest that high-quality, quan- 
titative evidence will persuade universities of ser- 
vice-learning’s pedagogical value and thus pro- 
mote greater acceptance (Bacon, 2001; Bringle & 
Hatcher, 2000; Bringle, Philips, & Hudson, 2004; 
Holland, Gelmon, Furco, & Bringle, 2005; 
Zlotkowski, 2000). Indeed, for those of us in the 
university, research characterizes “the kind of thing 
we do around here.” 1 However, “the kind of thing 
we do around here” is changing in profound ways 
and this change shapes our study of service-learn- 
ing classes and programs. To explicate the shifting 
context in which our research-based assessment 
occurred, this section outlines the ways that univer- 
sities are reconstituting knowledge-making activi- 
ties and reevaluating their roles in metropolitan 
communities. Considerations of the university’s 
relationship to its community have been present in 
rhetoric and composition research for some time, 
and several key studies have built on sophisticated 
notions of reciprocity and partnership. To further 
lay the groundwork for the control group study pre- 
sented in this article — a study that suggests that 
partnership-centered, community-based learning 
activities significantly enhance student writing 
skills — we will briefly introduce the Chicago Civic 
Leadership Certificate Program (CCLCP). We 
believe that the opportunity our program gives stu- 
dents to work within reciprocal community part- 
nerships is the key to understanding why our pro- 
gram creates an academic advantage. Therefore, 
our description of CCLCP will emphasize how rec- 


iprocal partnerships have contributed to its devel- 
opment. 

Both vernacular and academic sets of knowledge 
are contributing to new dialogues and unique per- 
spectives that are resulting in radical changes in 
knowledge production. A transdisciplinary 
(Gibbons et al. 1994, p. 3) approach to research 
that is demand-driven, methodologically flexible, 
entrepreneurial, collaborative, and embedded in 
complex contexts is emerging. For example, a 
University of Illinois at Chicago (UIC) professor 
has initiated the Chicago Public Art Group, which 
partners with “city agencies, private firms, and 
other organizations to produce community-orient- 
ed, site-integrated public artworks in which artists 
work with architects, designers, and engineers in 
the early planning stages” (Gude, 2000, p. 2). As 
part of its ongoing work, the art group seizes on a 
unique idea for developing a “place,” begins by 
creating a dialogue among all stakeholders, and 
then continues by conducting research, exploring 
the site, working collaboratively to create a budget, 
actually making the space, evaluating its use, and 
celebrating its presence. This exemplifies how 
knowledge is produced in its “context of applica- 
tion” (Gibbons et al., p. 3) and must be evaluated 
for its contribution to that context. The contrast is 
striking to traditional, university-based disciplinary 
activity, in which knowledge production responds 
to internally-driven, scholarly agendas. This 
reshaping of knowledge-making activities has 
upset the routine of researchers who raided off- 
campus communities for data, and of universities 
that participated in the destruction of thriving com- 
munities under the banner of urban renewal 
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(Muthesius, 2000; Wiewel & Broski, 1999). 2 In 
currently materializing higher-education contexts, 
partnership and reciprocity form the basis of par- 
ticipatory research that creates community assets 
while enriching the knowledge work of the metro- 
politan university (Walshok, 1995; see also 
Feldman, manuscript). Holland (2005) predicts that 
engaged research will become an important mea- 
sure of “academic quality and prestige.” As 
research activities in higher education don the man- 
tle of engagement, so do teaching and learning, 
often in the form of service-learning or communi- 
ty-based learning. Further, as Holland (2006) 
explains, changes in institutional mission also cre- 
ate consequences for teaching and learning. 
Universities have become interested in attracting 
students who want to participate as both knowledge 
workers and productive citizens in the emerging 
global context. Universities also are interested in 
improving teaching and learning outcomes, 
demonstrating accountability, and encouraging 
both civic-mindedness and professionalism in stu- 
dents (Holland). 

Keith (2005) characterizes the challenges of 
developing service-learning based on an under- 
standing of reciprocity that obviates the “server- 
served” relationship (p. 13; see also Feldman, 
2003). She sees productive opportunities for stu- 
dent learning when reciprocity “emphasizes 
respectful listening of perspectives and histories, 
together with community-building and possibly 
advocacy in an environment that acknowledges dif- 
ficult emotions and political choices that accompa- 
ny these tensions on both sides” (p. 15). What is 
needed, Keith argues, is “to rebuild discursive and 
public spaces in local communities” (p. 17). In 
other words, students should do more than partici- 
pate in direct service at local community-based 
agencies; they should participate in rethinking with 
partner agencies not only the quality of their ser- 
vices, but also the political and economic contexts 
in which services are delivered; and, finally, their 
roles in policy-making that contributes to change. 

In higher education, the agenda outlined above 
cannot be carried out by individual faculty mem- 
bers and small groups of students; enactment 
requires an institutional commitment. The faculty 
member who, with community partners, collabora- 
tively researches ways of providing low-cost hous- 
ing, planning for lead abatement, evaluating or 
improving public transportation, identifying com- 
munity assets, lessening the high-school dropout 
rate, or strengthening the capacity of not-for-prof- 
its, has at hand a ready-made, community-based 
context in which to introduce students to commu- 
nity-based learning. But if the faculty member’s 
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community-based research is not valued in the 
institution, neither will be the teaching and learning 
outgrowths of that scholarship. 

Community-based learning that arises from English 
studies and, in particular, from scholarship in rhetoric 
and composition, has generated a stream of research 
that focuses on the importance of reciprocal partner- 
ships for student learning. This research builds on a 
theoretical predisposition that defines writing, and lan- 
guage use in general, as a consequential, meaning- 
making activity (Bawarshi, 2003; Crosswhite, 1996; 
Mailloux, 1995; Petraglia, 2000; Russell, 1995). 
Reporting on her community-based literacy research, 
Cushman ( 1996) writes about the problems of slippery 
discourse, the challenge of taking into account the per- 
spectives of community members outside the acade- 
my, and the seductive nature of academic privilege. 
Cushman (2002) defines research-based service-learn- 
ing as an intellectually challenging activity that syn- 
thesizes research with both teaching and building 
community capacity. She supports the argument we 
want to make: theory-based pedagogical efforts con- 
nected to rigorous, ongoing research and, we would 
add, assessment, have a greater potential to be accept- 
ed as “what we do around here.” Linda Flower (2002) 
argues that the knowledge-making “we do around 
here” can be shifted to the broader landscape “out 
there.” For Flower, notions of reciprocal partnerships 
bring into focus the competing social, cultural, and lit- 
erary epistemologies at work when students enter a 
community. Flower argues for “transformative under- 
standing” as an active, participatory practice that sup- 
ports students as they move beyond good intentions to 
understand how one can use language as a form of 
praxis, that is, informed action (pp. 184-187). 

Through analyzing a series of case studies. 
Deans (2000) illustrates how student writing can 
move from decontextualized classroom activities to 
participatory activity that contributes to real-world 
change. His three models for community-based 
writing — writing for the community, writing about 
the community, and writing with the community — 
provided the framework for the wide range of writ- 
ing students do in the Chicago Civic Leadership 
Certificate Program (CCLCP). Much of the writing 
in typical service-learning courses constitutes what 
Deans calls “writing about” the community: stu- 
dents have an experience in the field and return to 
classroom to process and critique that experience 
in writing. Deans' case study of Bruce Herzberg’s 
two-semester sequence of writing courses at 
Bentley College describes a version of “writing 
about” informed by Herzberg’s dual commitment 
to social justice and to preparing students to com- 
prehend and produce academic discourse. 
Herzberg’s focus, Deans writes, is on critical peda- 
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gogy; he encourages students to critique “dominant 
social institutions (particularly schools) and domi- 
nant attitudes (particularly the ubiquitous 
American faith in individualism and meritocracy)” 
(p. 91). During Herzberg’s second-semester course, 
students produce lengthy research-based papers 
about important social issues that are intended for 
an academic audience and, particularly, for the 
teacher (Deans, p. 97). 

Service-learning’s standard rationale for “writ- 
ing about” is that learning occurs through a cycli- 
cal pattern of experience in the community fol- 
lowed by written reflection produced in the class- 
room (Eyler & Giles, 1999, p. 7, pp. 175-176). In 
classes such as Herzberg’s, “writing about” encour- 
ages students to explicate and support thoughtful 
critique in carefully-constructed academic argu- 
ments, but when students are asked to use writing 
only as a way of recording and/or expressing their 
personal reactions to experience, they may not 
come to understand and appreciate writing’s 
rhetorical power to give shape to situations and 
influence outcomes. In such cases, students easily 
misperceive writing as a mere conduit for carrying 
reports of experience to the teacher. We argue that 
writing does not reflect experience or feelings 
about experience as might a mirror; rather, writing 
is a tool for making meaning in social contexts. 
This premise has guided the CCLCP’s emphasis on 
involving students in reciprocal partnerships to 
produce writing for partner organizations. 

Program Design 

The Chicago Civic Leadership Certificate 
Program (CCLCP) offers incoming UIC students 
the opportunity to add to their undergraduate work 
a dimension of civic participation through the 
study of writing, rhetoric, and urban issues. 3 
CCLCP develops reciprocal partnerships with not- 
for-profit organizations during a curriculum plan- 
ning process that focuses on meeting partner orga- 
nizations’ needs for research assistance and for 
written documents in diverse genres. In the current 
pilot program, students are earning a certificate and 
qualifying for paid, community-based internships 
by taking five classes (one each semester). Each 
class includes fieldwork that requires students to 
apply and/or to test classroom learning in the con- 
text of projects undertaken with and for partner 
not-for-profit organizations. During 2004-05, the 
first year of the three-year pilot program, two 
semester-long writing courses provided a founda- 
tion in rhetorical analysis and argument. Students 
designed and wrote documents during fieldwork 
experience with local non-profits such as Gads Hill 
Center, The Resurrection Project, and Changing 


Worlds. During the second year of the pilot, 2005- 
06, students took two courses in UIC’s College of 
Urban Planning and Public Affairs, and worked on 
related research-based writing projects during 
fieldwork with nonprofit partners, including plan- 
ning organizations such as the Center for 
Neighborhood Technology, the Northeastern 
Illinois Planning Commission, and the Metro 
Chicago Information Center. The final course in 
the sequence, “English 375: Rhetoric and Public 
Life,” to be offered in Fall 2006, requires a cap- 
stone project, collaboratively designed by a student 
and a community partner of the student’s choice, 
and implemented by the student to the partner’s 
specifications. Students who earn the CCLCP cer- 
tificate may apply for paid internships with the pro- 
gram’s student-centered CityWorks Center, which 
will open in January 2007. Interns with CCLCP 
certificates will be very well-prepared to extend 
their work with community partners at local sites. 4 

Student learning in the CCLCP depends on writ- 
ing activity that is carried out, at least in part, in the 
context of reciprocal partnerships. Reciprocity, for 
CCLCP, refers to the complex relationships devel- 
oped among the program staff, faculty, students, 
and the not-for-profit organizations that partner 
with CCLCP for the purpose of producing written 
documents needed by the organization. This rela- 
tionship exemplifies a “community of practice” 
(Wenger, 1998), defined as “groups of people who 
share a concern, a set of problems, or a passion 
about a topic, and who deepen their knowledge and 
expertise in this area by interacting on an ongoing 
basis” (Wenger, McDerMott, & Snyder, 2002; p. 
34; see Keith, 2005, for a useful discussion of rec- 
iprocity). For instance, a student working to pro- 
duce a fact sheet for Gads Hill Center, a venerable 
social service organization and CCLCP partner, 
must become immersed in many aspects of the 
agency to design and produce the document. Such 
work transcends the “writing for” category in that 
it is reciprocal. In the process of working together 
to define needs and problem-solve, students and 
partners establish deep and ongoing relationships 
in which all participants contribute to social capi- 
tal. These relationships cannot be characterized as 
simple information exchanges; rather, the 
exchanges that transpire in a community of prac- 
tice build knowledge in the context of a joint, col- 
laborative venture fueled by the understanding that 
reciprocity depends on consequential, long-term 
social relationships. Communities of practice, like 
the ones we hope to create for CCLCP students, 
provide what Vygotskian learning theorists call a 
“zone of proximal development’” in which “social 
relations provide a motive and a context for apply- 
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ing and acquiring knowledge” (Moll & Greenberg, 
1990, p. 326). CCLCP students learn in situated 
contexts how writing works, and then can transfer 
that knowledge to other contexts in which they find 
themselves writing. 

Our commitment to developing reciprocal part- 
nerships influenced not only the design of the 
CCLCP but also our plans for assessment and 
research. It took months to lay the groundwork for 
CCLCP by developing partnerships with not-for- 
profit community organizations based on each part- 
ner organization’s needs for specific research 
and/or written documents. Throughout this 
process, we worked closely with UIC’s Great 
Cities Neighborhoods Initiative, which has a track 
record of creating and maintaining successful com- 
munity-university partnerships. With CCLCP com- 
munity partners as co-teachers, we collaboratively 
designed projects for students, who, immersed in 
the work of their organizations would go on to 
plan, design, and produce (mostly research-based) 
documents, needed by their community partner 
organizations. While classroom lessons concerned 
rhetoric, genre, effective writing, argumentation, 
and documented research, as well as urban theory 
and issues, students' lessons in the field invited a 
rich sense of participation in a community of prac- 
tice (Lave & Wenger, 1991; Wenger, 1998). 

What impact did the CCLCP students’ immer- 
sion in reciprocal community partnerships have on 
their writing? The next part of this article traces the 
design of an assessment matrix that crystallized 
student learning goals, guided syllabus develop- 
ment, and focused program evaluation on student 
learning outcomes. Our student learning outcomes 
matrix proved heuristic; questions it posed led to 
further questions about whether students’ participa- 
tion in the language activities of community-based 
partners would have an impact on the students’ 
academic writing. That question inspired the 
research we report in this article. Because CCLCP 
grew out of an established, university-wide, first- 
year writing program we had both the access and 
the set of writing criteria we needed to design a 
study comparing CCLCP students’ academic 
research papers to the papers of a control group of 
students who did not participate in a community- 
based learning experience. 

Assessment Matrix for Student Learning 
Outcomes 

Developing assessment matrices offered an 
important opportunity to translate CCLCP’s con- 
ceptual framework into clearly articulated, measur- 
able goals. Our matrices, modeled on work by 
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Gelmon, Holland, Driscoll, Spring, and Kerrigan 
(2001), address three areas: student learning, facul- 
ty development, and community partnership. 
Because this article focuses on the impact of recip- 
rocal partnerships on student learning, we will con- 
fine our discussion to the student learning out- 
comes matrix (see Table 1). The task of identifying 
specific learning goals and asking how we might 
know whether they had been met began early and 
continued simultaneously with syllabus develop- 
ment during the summer before CCLCP’s inaugur- 
al semester. Designing the learning outcomes 
matrix turned out to be a challenging, recursive 
process that tested our ability to translate theory 
into instructional activities that would support stu- 
dent achievement of CCLCP learning goals. 

The first column in CCLCP’s 2004-2005 matrix 
for student learning, labeled "We want to know...”, 
presents our program’s core concepts. The left side 
of that column categorizes the core concepts into 
the four main strands of CCLCP: rhetoric and aca- 
demics, community-based writing and research, 
civic engagement, and leadership. The right side 
frames our core concepts as a series of questions. 
The task of posing these questions demanded that a 
baggy set of theoretical beliefs be written as 
testable questions, research questions, if you will. 
The exercise brings home, especially to writing 
teachers, the rhetoric at work in articulating learn- 
ing goals for students. The second column, labeled 
"And we will know by...”, asks for key indicators of 
student accomplishment. The third column identi- 
fies sources of data. Each instructional activity 
planned for CCLCP’s writing classes would be 
tested against the learning goals identified in the 
assessment matrix and the goals would be shared 
with students so they would understand what we 
expected them to learn. 

In the previous section, we sketched the ways in 
which our community-based learning project built 
on an institution-wide commitment to engaged 
scholarship and on discipline-specific notions 
about how reciprocal partnerships enhance writing 
instruction. Learning goals for CCLCP students 
came alive through writing projects with real-world 
impact. The projects were shaped by ongoing reci- 
procal relationships with community partners and 
enacted lessons about how language functions 
rhetorically in particular contexts. All this hap- 
pened because of a question we asked ourselves 
about our matrix’s first two conceptual categories, 
rhetoric and academics, and community-based 
writing and research. This key question was, “But 
how will we know whether students apply their 
rhetorical lessons in community-based contexts?” 
As we realized, the answer to that question lay in 
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Table 1 


Assessment Matrix for Student Learning Outcomes 


We want to know...(Core Concepts) 

And we will know by ... (Key Indicators) 

Method 

Rhetoric and 
Academics 

How do students apply the 
rhetorical dimensions of 
situation, genre, language 
and consequences in the 
context of their work with 
at community agencies? 

Articulation and application of rhetoric to 
specific situations. 

Analysis of cover 
letters, field notes, 
other student writing; 
student focus group. 


How does writing and 
learning in this program 
facilitate writing and learning 
in other university classes? 

Articulation of how students’ understand- 
ing of rhetoric has facilitated writing and 
learning in other university classes. 

Course surveys; student 
focus group; field 
notes, and other student 
writing. 

Community -based 
writing and research 

How effectively are the 
documents created for the 
community partners? 

Feedback from community partners. 
Feedback from faculty. 

Student self-evaluation of changes in 
writing skills. 

Community partner 
focus group; faculty 
interviews; analysis of 
field notes, student 
surveys, and other 
student writing 


How do students apply 
communication methods 
and skills to conduct research 
within various discourse 
communities? 

Feedback from community partners. 
Feedback from faculty. 

Student self-evaluation of changes in 
research skills. 

Student focus group; 
student surveys; analy- 
sis of field notes and 
other student writing. 


How do students adjust their 
communication styles to 
enable them to function 
optimally in both complex 
community-based and 
university-based situations? 

Perceived change in ability to identify 
communication problems. 

Feedback from faculty. 

Feedback from community partners . 

Student focus group; 
analysis of field notes 
and other student writ- 
ing; faculty interviews; 
community partner 
focus group. 

Civic Engagement 

How do students value civic 
engagement for its impact on 
society at large? 

Articulation of knowledge about 
community and public issues. 
Recognition of changes or possibilities 
for change in public life as a result of 
their actions. 

Understanding of community strengths, 
problems, resources. 

Identification of community assets and 
needs. 

Analysis of research 
papers; student focus 
group. 


How do students integrate 
their civic engagement 
activities with their academic, 
career and personal goals? 

Choice of a major that enables students 
to pursue civic engagement activities. 
Career decisions that enable students to 
pursue civic engagement activities. 
Extra-curricular activities that enable 
students to pursue civic engagement 
activities. 

Student surveys; 
student focus group. 

Leadership 

How does participation in this 
program shape students’ 
understanding of leadership? 

Articulation of a dynamic definition of 
leadership that includes examples. 
Articulation of how leadership skills 
have been demonstrated by community 
partners. 

Student focus group; 
community partner 
focus group. 


How does participation in this 
program shape students’ 
ability to lead? 

Sense of responsibility for solving prob- 
lems and taking ownership of projects. 
Ability to collaborate with others and 
facilitate teamwork. 

Faculty interviews; 
community partner 
focus group; student 
focus group. 
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our examination of the brochures and fact sheets 
first-semester CCLCP students produced for com- 
munity partners; these documents demonstrate the 
extent to which classroom learning was enacted in 
a community context. 

The learning outcomes matrix also poses the 
question, “How do students adjust their communi- 
cation styles to enable them to function optimally 
in both complex community-based and university- 
based situations?” This question led us to examine 
students’ ability to accomplish both community- 
based writing for their partner organizations — the 
kind of writing they attempted when producing 
brochures and fact sheets in their first-semester 
writing course — and written academic discourse. 
We looked for evidence of competent academic 
writing in the research papers produced in English 
161, the second-semester writing course required 
of all UIC undergraduates, including those enrolled 
in CCLCP. 

Question for Research 

Our study investigated not only whether CCLCP 
students could competently generate an extended 
academic argument based on inquiry, but whether 
students’ community-based learning experience 
had an impact on the quality of their end-of semes- 
ter research papers. Our Lirst-Year Writing 
Program has established evaluation criteria for 
these research papers, so a panel of readers was 
able to refer to the same standard criteria when rat- 
ing all the papers in a mixed sample of papers writ- 
ten by students in CCLCP’s community-based 
learning version of English 161 and papers pro- 
duced by a control group of students in traditional 
English 161 courses. We then compared the aver- 
age scores of each group. We argued above that in 
addition to responding to calls for more rigorous 
research design, service-learning scholars should 
contemplate how to design research that captures 
the complexity of partnership-centered, communi- 
ty-based learning. The study presented here aims to 
test the effect of writing experiences situated in the 
context of reciprocal partnerships with not-for- 
profit organizations. 

Both the CCLCP students and control group stu- 
dents were enrolled during the same semester, 
Spring 2005, in sections of English 161. It is 
important to note that the conceptual framework 
was the same for both classes. English 161 requires 
all students to participate in academic culture by 
conducting guided research, posing questions 
about important issues, and developing an argu- 
ment in response to positions taken by others. We 
want students in this course to see themselves as 
participants in a consequential investigation, and to 
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position themselves within an ongoing conversa- 
tion about the topic of their research. Each English 
161 instructor develops a particular thematic 
inquiry and students may choose a class that suits 
their interests, although, of course, many students 
choose classes that best fit their schedules. Typical 
fields of inquiry for English 161 include immigra- 
tion, community development, animal rights, gen- 
trification, globalization, education, or issues cen- 
tering on race or ethnicity, class, and gender. 
During the first half of the semester, students in tra- 
ditional English 161 classes read about the class’s 
broad topic in a variety of disciplinary and vernac- 
ular genres, and produce short papers demonstrat- 
ing mastery of summary, synthesis, and analysis. 
Although English 161 students may explore their 
own communities or communities neighboring 
UIC by conducting interviews and examining 
archival materials, more typically their research is 
drawn from academic journals, scholarly books, 
and other material available though research 
libraries. 

Classroom activities in the CCLCP section of 
English 161 paralleled activities in the traditional 
sections. However, a key difference occurred in the 
projects produced in the first part of the semester. 
CCLCP students completed community-based 
writing projects that responded to needs identified 
by their partner organizations, such as portions of 
annual reports, press kits, brochures, Web content, 
and feature stories. CCLCP students learned about 
summary, synthesis, and analysis by applying these 
skills to analyzing both documents produced by 
their partner organizations and academic readings 
on rhetoric and urban issues. Because CCLCP stu- 
dents learned to apply the skills of summary, syn- 
thesis, and analysis from the perspective of their 
community partners, students came to understand 
exigence and what it means to respond rhetorically 
to a real-world situation. Summary, synthesis, and 
analysis were no longer decontextualized intellec- 
tual strategies, but practices taken up in response to 
an organization’s need to achieve usually concrete 
ends. 

From the beginning of the semester, just as in 
traditional English 161 classes, CCLCP students 
began to contemplate their research paper topics. 
Most students consulted with their community 
partners and chose an issue related to their partner 
organization’s work. A few students chose to write 
about service-learning itself, with a focus on the 
pedagogy of the CCLCP. About midway through 
the semester, instruction in both the CCLCP and 
traditional sections of English 161 turned to the 
origination of a research proposal. From that point, 
instruction increasingly focused on academic 



Imposed -MJCSL 13-1 10/11/06 


1:09 PM 


Page 



Feldman et al. 

inquiry and development of an argument-based, 
documented investigation that would result in a 10- 
to- 12-page research paper. For the past ten years or 
so, UIC’s First- Year Writing Program has relied on 
a criterion-based evaluation rubric for the research 
paper, developed collaboratively by members of 
the teaching staff. The rubric (see Table 2) identi- 
fies key course features that define program-wide 
instruction and evaluation. These criteria stem, at 
least in part, from the First- Year Writing Program’s 
commitment to engaged scholarship. The work 
expected of first-year writing students parallels — 
in a broad, conceptual way — the scholarly work of 
faculty who investigate significant issues in their 
own disciplinary contexts. As such, the features of 
the rubric emphasize the contextual, participatory 
features of research, and our program’s attempts to 
bring students into dialogue with authors and 
stakeholders concerned about the issue under 
study. Thus, our research paper evaluation rubric 
draws on the broad intellectual framework of 
engagement and sets the same goals for both tradi- 
tional and CCLCP students. “Engagement,” 
throughout the First- Year Writing Program, refers 
broadly to engagement with textual voices that 
have something to say about important civic and/or 
social issues. Instruction in CCLCP, we claim, with 
its strong focus on reciprocal partnership, adds an 
extra dimension and thereby offers a qualitatively 
different experience of engagement. 


With the introduction of these specific evaluation 
criteria, we can fine-tune our research question. We 
were able to ask to what degree, on average, each 
set of papers adhered to each of the five research 
paper criteria established by our First- Year Writing 
Program. 

Research Design 

We began by constructing a control group of 
research papers to compare to CCLCP papers. A 
pool of 70 English 161 students was chosen from 
the universe of 1,864 students enrolled in tradition- 
al English 161 classes during Spring Semester 
2005. None of the 70 students had been in classes 
taught by any of the four readers involved in this 
study. This pool was not constituted at random: 
CCLCP students had been recruited and admitted 
to the program based on ACT English subscores of 
22 or more, so the traditional-course pool was 
matched to the CCLCP group on the basis of that 
score and also for gender and ethnicity. This strat- 
egy helped establish a viable control group, albeit 
not a perfect one. We were unable to control for 
variables such as previous service-learning experi- 
ence, interest in community issues or volunteerism, 
high school preparation, second-language influ- 
ence, prior instruction and quality of instruction in 
college composition, socioeconomic status, and 
year in college. We also were unable to control for 
the fact that CCLCP students had applied to partic- 


Table 2 


Evaluation Criteria for English 161 Research Papers 


Evaluation Criterion 

Description 

Taking a position 

The writer articulates a position or thesis that contributes to a significant 
public conversation. The position relates to key themes discussed in the 
class materials and work. The writer attends to the consequences of his or 
her position, its personal relevance, and the potential or real public 
impact. 

Developing arguments in context 

The writer understands that arguments emerge from important public and 
academic conversations in which participants respond to each other as if 
in dialogue. They question claims, ask questions about evidence, consider 
the appropriateness of the evidence, qualify their assertions, and respond 
to counter claims. 

Using sources effectively 

The writer identifies and reviews appropriate source material relevant to 
his or her position, characterizes the sources’ arguments, discusses 
disciplinary methods and approaches, provides historical context, 
critiques the sources, and considers the sources’ perspectives. 

Engaging intellectual strategies 

The writer demonstrates the ability to engage in a dialogue of ideas with 
the sources used in the paper. The work is enhanced by the ability to 
summarize, synthesize, and analyze. In addition, writers demonstrate how 
appropriate paraphrasing and quoting contribute to this dialogue of ideas. 

Using language appropriately 

The writer makes grammar and stylistic choices appropriate to the 
audience and purpose. The writer also cites sources appropriately, 
integrating the cited material into the writer’s work. 
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ipate in the program and that the CCLCP cohort 
had “bonded” by studying writing together the pre- 
vious semester. For the research study, we decided 
that roughly 50 percent of the papers from CCLCP 
cohort of 33 students would be reviewed, so 16 
CCLCP papers were randomly selected for review. 
Sixteen control group papers were then randomly 
selected from the pool of 70 drawn from tradition- 
al English 161 classes. To prepare for the review, 
all identifying marks were removed from all the 
papers. Four English 161 instructors were 
employed to review the papers; three were graduate 
teaching assistants and one a teaching adjunct. 
Before teaching writing at UIC, the teaching assis- 
tants completed a semester-long graduate course 
on first-year writing pedagogy about the conceptu- 
al framework that drives writing instruction at UIC, 
designed syllabi for the courses they would teach, 
and practiced grading research papers using our 
program-wide evaluation criteria (see Table 2). The 
fourth reader, a full-time lecturer, had prepared to 
teach writing at UIC by attending intensive training 
sessions and developing detailed syllabi that were 
reviewed and approved by the First- Year Writing 
Program administration staff. While the instructors 
chosen to review papers for this study had signifi- 
cant and relevant training and experience, they had 
no connection to courses comprised by the CCLCP. 

The readers met over three consecutive days in 
mid- August 2005. A portion of the first day was 
spent preparing to score the papers. First, the read- 
ers discussed each evaluation criterion to come to a 
shared definition. Next, they participated in a cali- 
bration exercise, individually reading a sample 
paper and rating it on a scale of one through five on 
each of the five criteria. The group then discussed 
the scores. This exercise was intended to help read- 
ers develop a common understanding of how to 
apply the standards against which papers were to 
be scored. After this calibration exercise and 
throughout the next two days, each reader read 
each of the 32 papers, scoring each paper on each 
of the five criteria in the evaluation rubric. This cri- 
terion-based scoring method was introduced into 
writing assessment by Lloyd-Jones in his essay 
“Primary Trait Scoring” (1977), and is described 
by Bizzell (1987) in her discussion of later work by 
noted composition scholars Faigley, Cherry, 
Jolliffe, and Skinner (1985). As Bizzell explains, 
primary trait scoring “focuses on a set of criteria 
derived from the kind of task posed” to students 
(p. 580). This approach, Bizzell notes, provides 
much more information than other methods, but is 
extremely labor-intensive. 

During the three-day reading process two issues 
emerged. First, even though the 32 papers to be 
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read were presented as a single, mixed sample, 
meaning they were not grouped by origin, early in 
the reading process readers noticed that they were 
dealing with two distinct sets of papers. 
Specifically, they observed that in one set of papers 
the research topics were much more clearly defined 
than in the other. (The clearly defined topics, as it 
turned out, emerged from the CCLCP papers.) 
Despite this observation, the readers were encour- 
aged to continue applying the evaluation criteria to 
every paper, as they had been trained to do. A sec- 
ond, unanticipated, question arose about grading. 
Even through the evaluation criteria are used in 
every English 161 class, instructors frequently use 
them holistically, as a guide to assigning a single 
composite grade to each paper. Further discussion 
helped clarify that the readers should not imagine 
that they were scoring papers holistically, in the 
interest of arriving at a grade. Accordingly, they 
were instructed to use the full range of scores, one 
through five, to denote the degree to which a paper 
fulfilled each separate criterion. 

Results 

In this study we asked to what degree, on aver- 
age, each of the two sets of research papers — from 
the CCLCP class and from the traditional group- 
adhered to each of the five evaluation criteria for 
English 161. We found a statistically significant 
difference in each average primary trait score 
between the papers produced by CCLCP students 
and those written by students in traditional classes 
(see Table 3). This finding suggests that CCLCP 
students’ partnership-centered, community-based 
learning activities contributed to the relatively 
greater academic success of their final research 
papers. Inter-rater reliability, however, remains 
problematic in writing-related research on learn- 
ing, and we are considering measures to strengthen 
this feature of future studies. 

Research Paper Review 

On each of the five criteria, the average score 
assigned to research papers written by CCLCP stu- 
dents in English 161 exceeded the average score 
earned by papers written by students in traditional 
English 161 classes. Table 1 displays the mean raw 
score and level of significance for each set of 
papers on each of the five criteria, along with the F 
score (analysis of variance) on each criterion. The 
differences are not slight, yielding F-scores ranging 
from 20.70 to 35.88. Despite the difficulty with 
inter-rater reliability, discussed below, the robust 
level of difference between the CCLCP students’ 
papers and the traditional student papers suggest 
that the CCLCP students’ community-based learn- 
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Table 3 


Differences between Community-based and Traditional Writing Courses on 
Five Primary Evaluation Criteria 


Criterion 

n 

Mean 

F score 

Sig. (pc. 05) 

Developing Arguments in Context 

CCLCP 

16 

3.73 

35.88 

.000 

Traditional 

16 

2.42 


.000 

Engaging Intellectual Strategies 

CCLCP 

16 

3.72 

34.61 

.000 

Traditional 

16 

2.39 


.000 

Using Sources Effectively 

CCLCP 

16 

3.67 

28.78 

.000 

Traditional 

16 

2.52 


.000 

Taking a Position 

CCLCP 

16 

3.81 

22.39 

.000 

Traditional 

16 

2.56 


.000 

Using Language Appropriately 

CCLCP 

16 

3.92 

20.70 

.000 

Traditional 

16 

3.09 


.000 


ing experience contributed to their successful 
acquisition of an academic skill that will have long- 
lasting impact on their work in a wide variety of 
academic and professional contexts. 

The contrast between the two sets of papers was 
so striking that we may assert that the decisive dif- 
ference was CCLCP students’ engagement in reci- 
procal partnerships developed with CCLCP partner 
organizations. As we argue above, reciprocal rela- 
tionships result from complex, long-term associa- 
tions that produce social capital through purposeful 
collaboration. The CCLCP experience differs from 
those service-learning activities in which students 
assist a partner organizations’ service consumers 
and then return to the classroom to use writing to 
reflect on the experience. Much of the CCLCP stu- 
dents’ writing occurred in the “context of applica- 
tion” (Gibbons, et al. 1994, p. 3) and thus emerged 
from the exigency of a particular situation. Situated 
learning activities based on the reciprocal relation- 
ships enabled by CCLCP added another layer to 
the typical English 161 experience. Presumably, 
students in all sections of English 161 study and 
write about consequential civic and/or social 
issues. Students in traditional classes even may 
venture into communities to gather first-hand infor- 
mation through interviews or obtain primary- 
source documents. However, as noted by the read- 
ers who participated in this study, CCLCP student 
writers’ strong sense of involvement with the issues 
they investigated helped those students construct 


cogent, well-supported arguments for clearly artic- 
ulated positions on “their” issues. 

Inter-rater Reliability 

We expected the calibration exercise used with 
the raters to produce an acceptable level of inter- 
rater reliability (Shohamy, Gordon, & Kraemer, 
1992, p. 31), but inter-rater reliability was lower 
than we’d hoped, and inconsistent across primary 
trait categories (ranging from .050 to .4159). The 
alphas were calculated using Krippendorf’s alpha, 
a test that measures inter-rater agreement while 
accounting for error related to chance (see Table 4). 

We compensated for low inter-rater reliability to 
some extent by averaging raters’ scores (Hartman, 
1982). There are other strategies as well: Wurr 
(2002), in a study of the impact of service-learning 
on writing skill, used fewer raters and employed an 
adjacency adjustment to achieve higher reliability 
by considering adjacent scores to be in agreement. 
In addition, in Wurr’s study, when raters’ scores 
were two or more points apart, the paper was sub- 
mitted to a third reader’s evaluation, and the three 
ratings were then averaged (p. 53). The low level of 
inter-rater reliability achieved in our study under- 
lines the axiom that reader interpretation is both an 
important and hard-to-control variable, contribut- 
ing to the great difficulty of assessing writing qual- 
ity in any context. Some researchers argue that it is 
not only problematic, but also unrealistic to expect 
qualitative studies to meet the standards of inter- 



Imposed -MJCSL 13-1 10/11/06 


1:09 PM Page 25 


Impact of Partnership-Centered, Community-Based Learning 


Table 4 

Inter-rater Reliability Alphas using Krippendorf’s Alpha 



CCLCP 

Traditional 

Taking a position 

.1977 

.4159 

Developing arguments in context 

.0770 

.2643 

Using sources effectively 

.0505 

.1722 

Engaging intellectual strategies 

.0774 

.3809 

Using language appropriately 

.1123 

.1302 


rater reliability used in quantitative studies (Harris, 
Pryor, & Adams, 1997). In “Can There Be Validity 
Without Reliability?” Moss (1994) challenges the 
assumption that a rigid notion of reliability is 
essential to assessment by advocating that we 
broaden our sometimes too-narrow psychometric 
understanding of reliability to include considera- 
tion of the broader context in which knowledge is 
being created. Nevertheless, we are seeking defen- 
sible methodological modifications, such as those 
used by Wurr, that might boost the inter-rate relia- 
bility of our next comparative study of writing 
quality. 

Discussion 

This study examines an elemental artifact of uni- 
versity student writing, the academic research 
paper, to draw conclusions about the impact of a 
partnership-centered, community-based learning 
experience. These data, 32 academic research 
papers, offered a way to study comparable writing 
products across different instructional settings. A 
wide variety of additional data exists that may, 
upon analysis, tell us more about CCLCP students’ 
experience. For example, CCLCP students wrote 
field notes about their experiences, produced pro- 
ject cover letters for their teachers, filled out sur- 
veys, participated in focus groups, designed and 
produced documents for and with their community 
partners, and wrote argumentative essays about 
what they had learned. As CCLCP moves ahead, 
these data are being explored as suggested by our 
assessment matrix. 

The research papers, themselves, reveal some 
features of what CCLCP students learned. The 
readers were called back together several weeks 
after the scoring exercise to “gloss,” that is, to 
interpret, their scores on a pair of research papers. 
This follow-up conversation was intended to illu- 
minate what qualities, precisely, distinguished the 
CCLCP papers from those produced by students in 
traditional classes. It is important to note that each 
of the two papers selected for discussion scored as 
average in respect to its group. The CCLCP stu- 


dent’s paper critiques his/her community partner’s 
use of statistics in its public relations documents, 
representing an atypical case of a student who was 
not “on board” with the agency’s approach to pub- 
licizing its mission. The student paper from the tra- 
ditional English 161class focuses on the Holocaust. 
We do not have further information on the broad 
inquiry of the course because identifying marks 
were removed for anonymity’s sake. For the same 
reason, we will refer to the community agency con- 
sidered in the CCLCP paper as the “not-for-profit” 
(NFP). We include in Table 5 representative read- 
ers’ comments relative to the first two research 
paper evaluation criteria. For each of these two cri- 
teria, we indicate the subject paper’s average score 
and a typical response from one of our four readers. 
Beneath the reader’s response, we include a section 
from the student text that exemplifies the rationale 
for the comment. 

In commenting on Criterion 1: Taking a Position, 
a reader points out that the CCLCP paper makes an 
arguable claim and answers the “so what?” ques- 
tion, announcing why it matters if the writer’s com- 
munity partner misuses statistics in its public rela- 
tions documents. The student paper from the tradi- 
tional class proposes that the Holocaust teaches a 
lesson that is “needed to be learned and remem- 
bered,” but never actually pursues or supports that 
claim. A reader points out, in discussing Criterion 
2: Developing Arguments in Context, that the 
CCLCP paper guides the reader by establishing a 
meta-structure that the reader can track. The reader 
also points out that the writer needs to include 
more counter-arguments. (The counter-argument, 
or “naysayer” voice, is a challenging dialogic move 
that identifies what others might say about the 
writer’s argument.) Students who learn in real- 
world contexts in which issues are in constant play 
will be more likely to recognize arguments and 
counter-arguments when applying classroom 
lessons in actual communities of practice. The stu- 
dent paper from the traditional class, unable to 
marshal supporting evidence, is trapped, if you 
will, by a personal story that she cannot connect to 
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Table 5 


Comparison of Two Average Research Papers 


Traditional Paper 
"Remembering the Holocaust” 

CCLCP Paper 

“Interpreting Statistics: A Rhetorical Analysis of the 
NFP’s Marketing Documents” 

Criterion 1: Taking a Position: The writer articulates a position or thesis that contributes to a significant public 
conversation. The position relates to key themes discussed in the class materials and work. The writer attends to the 
consequences of his or her position, its personal relevance, and the potential or real public impact. 

Mean Score: 2.56 

Mean Score: 3.81 

"Remembering the Holocaust” 

“Interpreting Statistics” 

Reader’s comments: ...no one will disagree with it... 
lessons “need to be learned and remembered” not dis- 
cussed in paper. 

Reader’s comments ...clearly arguable thesis... joins a 
conversation... potentially relevant for the reader. 

From the paper: “The American tragedy of September 1 1 
has made us aware that a single deed can obliterate from 
the face of the earth the civilization that has been built over 
the centuries, and that ordinary bacteria (such as anthrax) 
can strike fear into a population of millions. The tragedy 
of the Holocaust should be one of the biggest lessons that 
needed to be learned and remembered.” 

From the paper: “To the point, the NFP displays vague 
results, incorrect statistics, and dismisses important 
information in their documents. As a result, the public 
may make less than a knowledgeable decision about 
whether they want to work with the agency. The goal of 
this analysis is to demonstrate the government funded 
agencies should not misinform the public about the effect 
of the work they complete because they are taking money 
from the state.” 

Criterion 2: Developing Arguments in Context: The writer understands that arguments emerge from important 
public and academic conversations in which participants respond to each other as if in dialogue. They question 
claims, ask questions about evidence, consider the appropriateness of the evidence, qualify their assertions, and 
respond to counter claims. 

Mean Score: 2.42 

Mean Score: 3.73 

"Remembering the Holocaust” 

"Interpreting Statistics” 

Reader’s comments: “. . .glut of narrative because it isn’t 
used in service to the argument, although narrative 
certainly could. . .Because the thesis isn’t very powerful, 
the writer doesn’t show how it might not be simplistic. . .” 

Reader’s comments: “Intro established a problem, it told 
how it was going to talk about the problem. Every topic 
sentence had its own part of the introduction. . . around 
page 5 ... a very, very sophisticated rhetorical analysis... 
didn’t bring in enough counterarguments, but argument is 
clearly extended through the paper.” 

From the paper: “I have written a few essays on the 
Holocaust over the years. When the opportunity arises for 
me to write about history, the Holocaust automatically 
comes to mind.” 

From the paper: “NFP uses false statistics as a means of 
manipulating the public to believe in the work that they 
do on local economic development.” 



an ongoing, public conversation. His/her class, we 
surmise, did not provide him/her with the recipro- 
cal relationships within which an argument might 
come to life as a vibrant response within a commu- 
nity of practice. We wish to offer several caveats 
here: First, our argument for participant-centered, 
community-based learning should not be taken as 
an argument that such learning never can happen in 
courses confined to classrooms. We do not wish to 
create a false binary between on-campus and off- 
campus locations for learning. Rather, as Deans (in 
press) puts it, we wish to focus on the activity 
undertaken in any particular location. Along these 
same lines, our critical comments about the student 
paper on the Holocaust should not suggest that this 


topic could not be successfully handled as a contri- 
bution to any of a variety of ongoing discussions. 

Concluding Remarks 

Much work remains to be done. Even so, this 
research project can be seen as the first layer of 
excavation into a site replete with rich resources for 
gauging and understanding the impact of partner- 
ship-centered, community-based learning. The 
next iteration of CCLCP will allow us to conduct a 
similar study with a more diverse group of service- 
learning students than those participating in the 
current pilot program. And, because we plan to 
include in the next iteration both a fall and a spring 
semester English 161 class, well be able to com- 
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pare two groups of research papers without the 
possible interference of a “cohort effect.” We plan 
to review our options for achieving higher inter- 
rater reliability as well. As students proceed 
through the four courses that will be comprised by 
future versions of the CCLCP, we will be able to 
dig deeply into the questions suggested by our 
assessment matrix in search of the means to further 
characterize student learning. 

As important, we are eager to explore the chal- 
lenging questions posed by our assessment matri- 
ces about community partner and academic faculty 
outcomes. Many questions exist about the conse- 
quences, both intended and unintended, of the 
CCLCP program for not-for-profit, community- 
based organizations. Likewise, as more faculty 
become involved, we’d like to explore the link 
between our faculty’s engaged research with part- 
ner organizations and participant-centered, com- 
munity-based learning for undergraduates. Digging 
even deeper into our site, we can ask questions 
about the institutional home of CCLCP. Ever cog- 
nizant of UIC’s shifting history of community rela- 
tions — sometimes ironically, sometimes fittingly 
enacted on the actual site of Jane Addam’s Hull 
House — we wish to explore the institutional 
change precipitated by the CCLCP, the first under- 
graduate program devoted to community-based 
learning on a campus that, since 1993, has carried 
out its engagement activities through its Great 
Cities program of faculty research and scholarship, 
research centers, economic development, and 
neighborhood initiatives. 

The student learning outcomes we report in this 
article depend on the reciprocal partnerships devel- 
oped as part of the founding conception of the 
CCLCP. We argue that the learning that transpired 
for CCLCP students, writing, as they did, in com- 
plex social spaces driven by real-world exigency 
contributed to the more cogent arguments found in 
their research papers. This finding has had an insti- 
tutional impact at UIC. It signals to other faculty 
and administrators that partnership-centered, com- 
munity-based learning, and research into its out- 
comes is exactly the kind of thing we should be 
doing around here. 

Notes 

1 This phrase, “the kind of thing we do around here” 
and variations of it, such as “what we do around here” 
have been used to characterize the traditional work of 
academic departments. Notably, literary scholar and pub- 
lic intellectual Stanley Fish used the term to name the 
kind of work that should be done in English departments 
(Fish, 1995/1999; see also Olson, 2002). 

2 Most universities that have been present in urban 
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neighborhoods do not come to these new understandings 
from a pristine past. UIC, like other urban universities, has 
had a complex and sometimes contentious historical rela- 
tionship with its surrounding neighborhoods and efforts at 
engagement must overcome these historical difficulties. 

3 The Chicago Civic Leadership Certificate Program 
(CCLCP) pilot is funded by a three-year grant from the 
Coiporation for National and Community Service’s Learn 
and Serve America program, with an equal match from the 
University of Illinois at Chicago. Institutional Review 
Board permission was sought and awarded for research on 
this project under protocol 2004-0361, “Civic Leadership 
and Service.” CCLCP’s pilot has been generously support- 
ed by UIC Vice Provost for Undergraduate Studies Lon 
Kaufman, Dean of Liberal Arts and Sciences Christopher 
Comer, English Department Head Walter Benn Michaels, 
Dean of Urban Planning and Public Affairs Robin 
Hambleton, Great Cities Institute Director David Perry, 
and UIC Neighborhoods Initiative Associate Director 
Atanacio Gonzales. 

4 Future iterations of the Chicago Civic Leadership 
Certificate Program will include four courses instead of 
five. We learned through experience that students should 
complete the course sequence during their first two 
undergraduate years, creating more opportunity for their 
involvement with the soon-to-be implemented 
CityWorks Center. 
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